{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ab3c0f7f",
   "metadata": {},
   "source": [
    "## Конфигурация"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7663f7ca",
   "metadata": {},
   "source": [
    "Сначала зададим параметры приложения.\n",
    "\n",
    "Ниже примеры конфигурации для AWS EMR и для локального использования."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9d18d58",
   "metadata": {},
   "source": [
    "Конфигурация для AWS EMR.\n",
    "\n",
    "1. Добавим пакет `catboost-spark` с нужными версиями Spark, Scala и версией самого пакета.\n",
    "2. `deployMode` - `cluster`, потому что на driverе тоже делается нетривиальная работа, требующая ресурсы.\n",
    "3. Число vCpu на машинах `executor` ов = `spark.executor.cores` = `spark.tasks.cpus`, чтобы workerы CatBoost не мешали друг другу в одном процессе\n",
    "4. Нужно добавить не-JVM памяти (* .memoryOverhead) для нативного кода CatBoost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "05531d7a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Current session configs: <tt>{'conf': {'spark.jars.packages': 'ai.catboost:catboost-spark_3.0_2.12:0.25', 'spark.submit.deployMode': 'cluster', 'spark.driver.cores': 2, 'spark.driver.memory': '2g', 'spark.driver.memoryOverhead': '2g', 'spark.executor.memory': '2g', 'spark.executor.memoryOverhead': '2g', 'spark.executor.cores': 2, 'spark.tasks.cpus': 2}, 'proxyUser': 'user_ermakovpetr', 'kind': 'pyspark'}</tt><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "No active sessions."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%configure -f\n",
    "{\n",
    "    \"conf\": {\n",
    "        \"spark.jars.packages\": \"ai.catboost:catboost-spark_3.0_2.12:0.25\",\n",
    "        \"spark.submit.deployMode\": \"cluster\",\n",
    "        \"spark.driver.cores\": 2,\n",
    "        \"spark.driver.memory\": \"2g\",\n",
    "        \"spark.driver.memoryOverhead\": \"2g\",\n",
    "        \"spark.executor.memory\": \"2g\",\n",
    "        \"spark.executor.memoryOverhead\": \"2g\",\n",
    "        \"spark.executor.cores\": 2,\n",
    "        \"spark.tasks.cpus\": 2\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b994ff8b",
   "metadata": {},
   "source": [
    "Конфигурация для локального кластера.\n",
    "\n",
    "1. Добавим пакет `catboost-spark` с нужными версиями Spark, Scala и версией самого пакета.\n",
    "2. Число vCpu на машинах `executor` ов = `spark.executor.cores` = `spark.tasks.cpus`, чтобы workerы CatBoost не мешали друг другу в одном процессе"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "724b781d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PYSPARK_PYTHON=python3.6\n",
      "env: PYSPARK_DRIVER_PYTHON=python3.6\n"
     ]
    },
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'pyspark'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-509fa5c81ee2>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_line_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'env'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'PYSPARK_DRIVER_PYTHON=python3.6'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mpyspark\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msql\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mSparkSession\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m spark = (SparkSession.builder\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'pyspark'"
     ]
    }
   ],
   "source": [
    "%env PYSPARK_PYTHON=python3.6\n",
    "%env PYSPARK_DRIVER_PYTHON=python3.6\n",
    "\n",
    "from pyspark.sql import SparkSession\n",
    "\n",
    "spark = (SparkSession.builder\n",
    "    .master(\"local[*]\")\n",
    "    .config(\"spark.jars.packages\", \"ai.catboost:catboost-spark_3.0_2.12:0.25\")\n",
    "    .config(\"spark.executor.cores\", \"2\")\n",
    "    .config(\"spark.task.cpus\", \"2\")     \n",
    "    .appName(\"CatBoost_Spark_3\")\n",
    "    .getOrCreate()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3b0aaa9",
   "metadata": {},
   "source": [
    "Проверим SparkSession объект"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "848b5df0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f602bf6ffe248928a697219bf05410e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<pyspark.sql.session.SparkSession object at 0x7f04336c2150>"
     ]
    }
   ],
   "source": [
    "spark"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef952083",
   "metadata": {},
   "source": [
    "Импортируем `tempfile` - пригодится для создания временных директорий при обучении"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f7d3e0f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c9e27ec249f42cd96f44dc58cfd0c78",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import tempfile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ce5f019",
   "metadata": {},
   "source": [
    "Импорты нужных объектов из PySpark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a0c5da3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da8d37202ed14256b067b1f8cb8e4ec3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pyspark.ml.linalg import Vectors, VectorUDT\n",
    "\n",
    "from pyspark.sql import Row\n",
    "from pyspark.sql.types import *\n",
    "\n",
    "from pyspark.ml.linalg import *\n",
    "from pyspark.ml.param import *\n",
    "from pyspark.sql import *\n",
    "from pyspark.sql.types import *\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69d5ea55",
   "metadata": {},
   "source": [
    "Импортируем `catboost_spark`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "54c47d0a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dccaacee65d54e439056ae89bed59ee6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import catboost_spark"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee449242",
   "metadata": {},
   "source": [
    "## Задание схемы данных\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "810af055",
   "metadata": {},
   "source": [
    "Функция `createSchema` поможет создать нужную схему с метаданными для `DataFrames`, которые будет использовать CatBoost, в частности позволяет задать имена признаков и специфицировать категориальные признаки (если есть)\n",
    "\n",
    "Метаданные задаются словарем, передаваемым в параметр metadata конструктора `StructField`.\n",
    "Для численных признаков заполняем поле `numeric`, для категориальных - `nominal`.\n",
    "Для категориальных можно либо просто задать число уникальных значений (`num_vals`) либо еще указать их строковые представления (`vals`) (если они были получены из исходно строковых данных)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "65ee66fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7a41cfa0c2f4466962d3a7fd4d6d59b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def createSchema(\n",
    "    schemaDesc, #Seq[(String,DataType)],\n",
    "    featureNames, #Seq[String],\n",
    "    addFeatureNamesMetadata = True, # Boolean = true,\n",
    "    nullableFields = [], # Seq[String] = Seq(),\n",
    "    catFeaturesNumValues = {}, # Map[String,Int] = Map[String,Int](),\n",
    "    catFeaturesValues = {} #: Map[String,Seq[String]] = Map[String,Seq[String]]()\n",
    "): #: Seq[StructField] = {\n",
    "    result = []\n",
    "    for name, dataType in schemaDesc:\n",
    "        if (addFeatureNamesMetadata and (name == \"features\")):\n",
    "            numericAttrs = []\n",
    "            nominalAttrs = []\n",
    "\n",
    "            for i, fname in enumerate(featureNames):\n",
    "                if fname in catFeaturesNumValues:\n",
    "                    nominalAttrs.append({\"num_vals\": catFeaturesNumValues[fname], \"idx\": i, \"name\": fname})\n",
    "                if fname in catFeaturesValues:\n",
    "                    nominalAttrs.append({\"vals\": catFeaturesValues[fname], \"idx\": i, \"name\": fname})\n",
    "                else:\n",
    "                    numericAttrs.append({\"idx\": i, \"name\": fname})\n",
    "\n",
    "            attrs = {}\n",
    "            if numericAttrs:\n",
    "                attrs[\"numeric\"] = numericAttrs\n",
    "            if nominalAttrs:\n",
    "                attrs[\"nominal\"] = nominalAttrs\n",
    "\n",
    "            metadata = {\"ml_attr\": {\"attrs\": attrs, \"num_attrs\": len(featureNames)}}\n",
    "\n",
    "            result.append(\n",
    "                StructField(name, dataType, name in nullableFields, metadata)\n",
    "            )\n",
    "        else:\n",
    "            result.append(StructField(name, dataType, name in nullableFields))\n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8a9d37",
   "metadata": {},
   "source": [
    "## Простое обучение"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81284cc9",
   "metadata": {},
   "source": [
    "Создадим данные программно"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a304b845",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d08da0d910e4b64a167d695d570c6e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "featureNames = [\"f1\", \"f2\", \"f3\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "480804dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c337290349da44219cad52afc101948b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "srcDataSchema = createSchema(\n",
    "    [\n",
    "        (\"features\", VectorUDT()),\n",
    "        (\"label\", DoubleType())\n",
    "    ],\n",
    "    featureNames,\n",
    "    addFeatureNamesMetadata=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9e86195f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35f96fc0b0b249c28f6d33c99a9a99bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "srcData = [\n",
    "    Row(Vectors.dense(0.1, 0.2, 0.11), 1.0),\n",
    "    Row(Vectors.dense(0.97, 0.82, 0.33), 2.0),\n",
    "    Row(Vectors.dense(0.13, 0.22, 0.23), 2.0),\n",
    "    Row(Vectors.dense(0.14, 0.18, 0.1), 1.0),\n",
    "    Row(Vectors.dense(0.9, 0.67, 0.17), 2.0),\n",
    "    Row(Vectors.dense(0.66, 0.1, 0.31), 1.0)\n",
    "]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "df0e224e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5661ed20459f4f2b92bc136e43c6888e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6f4ab02f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e82bf61638d4cc09e9fd6c7986a9633",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------+-----+\n",
      "|        features|label|\n",
      "+----------------+-----+\n",
      "|  [0.1,0.2,0.11]|  1.0|\n",
      "|[0.97,0.82,0.33]|  2.0|\n",
      "|[0.13,0.22,0.23]|  2.0|\n",
      "| [0.14,0.18,0.1]|  1.0|\n",
      "| [0.9,0.67,0.17]|  2.0|\n",
      "| [0.66,0.1,0.31]|  1.0|\n",
      "+----------------+-----+"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d823008f",
   "metadata": {},
   "source": [
    "Создадим простой классификатор с 20 итерациями"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f6d34ad8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b4ff836ee9443e48f3d844a541ef33e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "classifier = (catboost_spark.CatBoostClassifier()\n",
    "      .setIterations(20)\n",
    "      .setTrainDir(tempfile.mkdtemp(prefix='test1')))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4173c3c8",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Обучим его, получим модель"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "befe4ff8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95f3997c6e1f4d218bc52ba118c168de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = classifier.fit(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec630f21",
   "metadata": {},
   "source": [
    "Модель можно применить на `DataFrame` с помощью функции `transform`. В результате получим `DataFrame` - копию исходного с дополнительными колонками.\n",
    "По умолчанию для классификатора будут доступны `rawPrediction`, `prediction` и `probability`, но можно выбрать необходимые с помощью методов `setRawPredictionCol`, `setPredictionCol`, `setProbabilityCol`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7b303e66",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f4781bd05b64bfc846946c36dd68020",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predDf = model.transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "21c66de3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39d68664e97e410e8036b99687ca0c2d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(features=DenseVector([0.1, 0.2, 0.11]), label=1.0, rawPrediction=DenseVector([0.0841, -0.0841]), probability=DenseVector([0.542, 0.458]), prediction=0.0)"
     ]
    }
   ],
   "source": [
    "predDf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ca6e40d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(features=DenseVector([0.1, 0.2, 0.11]), label=1.0, rawPrediction=DenseVector([0.0841, -0.0841]))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.setRawPredictionCol(\"rawPrediction\").setProbabilityCol(\"\").setPredictionCol(\"\")\n",
    "predDf = model.transform(df)\n",
    "predDf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6afc449a",
   "metadata": {},
   "source": [
    "Параметры можно также менять у уже готового объекта-Predictor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b82ebb74",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d8a7d50f09b4ccdbc999aa83010841f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(features=DenseVector([0.1, 0.2, 0.11]), label=1.0, rawPrediction=DenseVector([0.0159, -0.0159]), probability=DenseVector([0.5079, 0.4921]), prediction=0.0)"
     ]
    }
   ],
   "source": [
    "classifier.setLossFunction(\"Logloss\").setRsm(0.2)\n",
    "model = classifier.fit(df)\n",
    "predDf = model.transform(df)\n",
    "predDf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b314e13",
   "metadata": {},
   "source": [
    "## Сериализация"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf2a5130",
   "metadata": {},
   "source": [
    "Сам объект - `Predictor` (`CatBoostClassifier` или `CatBoostRegressor`) можно сериализовывать.\n",
    "Это удобно для сохранения настроек обучения."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7397431c",
   "metadata": {},
   "source": [
    "Сохраним объект на AWS в хранилище S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b4352806",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0f4e0bcc95874c84845caec97b558eec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "classifier.write().overwrite().save(\"s3a://aws-emr-resources-957875663655-us-east-2/my_classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d41f5ecc",
   "metadata": {},
   "source": [
    "Для локального режима используем простой путь"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b6f28746",
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier.write().overwrite().save(\"my_classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b221e3a6",
   "metadata": {},
   "source": [
    "Загрузим сохраненный объект из S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "372c8e4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e59c0046083745598708e2c2600d0d8b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CatBoostMLReader._java_loader_class.  ai.catboost.spark.CatBoostClassifier"
     ]
    }
   ],
   "source": [
    "loaded_classifier = catboost_spark.CatBoostClassifier.load(\"s3a://aws-emr-resources-957875663655-us-east-2/my_classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eff193ea",
   "metadata": {},
   "source": [
    "Загрузим локально сохраненный объект"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1f8e6ad8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CatBoostMLReader._java_loader_class.  ai.catboost.spark.CatBoostClassifier\n"
     ]
    }
   ],
   "source": [
    "loaded_classifier = catboost_spark.CatBoostClassifier.load(\"my_classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fce4d98",
   "metadata": {},
   "source": [
    "Обучим с помощью загруженного классификатора."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "19f19e70",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c6f3748b7e8344ceabff7ce2578dadd2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_from_loaded = loaded_classifier.fit(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d56a063",
   "metadata": {},
   "source": [
    "Модель тоже, естественно, можно сериализовать. Притом формат совместим с локальным CatBoost. Можно скачать сохраненную модель и использовать в применении уже с локальными библиотеками CatBoost. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "718a68b4",
   "metadata": {},
   "source": [
    "С использованием AWS S3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "0e43ef03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d25dbb63ac84f85a6a2b6232032f9b4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.write().overwrite().save(\"s3a://aws-emr-resources-957875663655-us-east-2/my_binclass_model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b75e3dbc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb3c0e6995294d58b2130632e22ac711",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CatBoostMLReader._java_loader_class.  ai.catboost.spark.CatBoostClassificationModel"
     ]
    }
   ],
   "source": [
    "loaded_model = catboost_spark.CatBoostClassificationModel.load(\"s3a://aws-emr-resources-957875663655-us-east-2/my_binclass_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba3c40f1",
   "metadata": {},
   "source": [
    "В локальном случае:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5f6475ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.write().overwrite().save(\"my_binclass_model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "67951228",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CatBoostMLReader._java_loader_class.  ai.catboost.spark.CatBoostClassificationModel\n"
     ]
    }
   ],
   "source": [
    "loaded_model = catboost_spark.CatBoostClassificationModel.load(\"my_binclass_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e95948e",
   "metadata": {},
   "source": [
    "Проверим что загруженная модель работает и результат тот же что и был на исходной"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "aedc922e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d26fb98ff3964d98a04db43fbf503aba",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(features=DenseVector([0.1, 0.2, 0.11]), label=1.0, rawPrediction=DenseVector([0.0159, -0.0159]), probability=DenseVector([0.5079, 0.4921]), prediction=0.0)"
     ]
    }
   ],
   "source": [
    "predDf = loaded_model.transform(df)\n",
    "predDf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fa24d21",
   "metadata": {},
   "source": [
    "## Тип Pool\n",
    "\n",
    "CatBoost у удобно иметь дополнительный тип датасета по сравнению и обычным DataFrame.\n",
    "Он является надстройкой над DataFrame.\n",
    "\n",
    "Зачем он нужен?\n",
    "1. Позволяет добавить в датасет дополнительные данные, такие как пары.\n",
    "2. Позволяет хранить преквантованные данные (об этом ниже).\n",
    "3. Удобнее для внутренного использования (API), хотя это не так важно пользователям.\n",
    "\n",
    "Создать Pool из DataFrame очень легко:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c5f14a5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "859ee682e50e4c999a508eac0c3158dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainPool = catboost_spark.Pool(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "371ca7a5",
   "metadata": {},
   "source": [
    "Исходный DataFrame хранится в атрибуте `data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "db77e04a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5036fc902d9444758af2b7d23f15cd96",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DataFrame[features: vector, label: double]"
     ]
    }
   ],
   "source": [
    "trainPool.data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbd0b3ec",
   "metadata": {},
   "source": [
    "## Обучение при наличии валидационных датасетов\n",
    "\n",
    "Теперь рассмотрим обучение при наличии валидационных датасетов. Их может быть несколько, на чаще всего один. На нем можно отслеживать целевые метрики и использовать overfitting detector.\n",
    "\n",
    "Использование валидационных датасетов пока возможно только когда датасеты имеют тип Pool. Для этого нужно передать их список вторым параметром функции `fit`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "804dd7b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b08478091a9e4aa6981f30654b934048",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "srcEvalData = [\n",
    "  Row(Vectors.dense(0.0, 0.33, 1.1), 1.0),\n",
    "  Row(Vectors.dense(0.02, 0.0, 0.38), 2.0),\n",
    "  Row(Vectors.dense(0.86, 0.54, 0.9), 1.0)\n",
    "]\n",
    "evalDf = spark.createDataFrame(spark.sparkContext.parallelize(srcEvalData), StructType(srcDataSchema))\n",
    "evalPool = catboost_spark.Pool(evalDf)\n",
    "model = classifier.fit(trainPool, [evalPool])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "f6242d61",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "84099ac97861433d9f1a32caa25fb0c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(features=DenseVector([0.0, 0.33, 1.1]), label=1.0, rawPrediction=DenseVector([0.0061, -0.0061]), probability=DenseVector([0.5031, 0.4969]), prediction=0.0)"
     ]
    }
   ],
   "source": [
    "model.transform(evalPool.data).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b181a739",
   "metadata": {},
   "source": [
    "Особенностью архитектуры распределенного CatBoost на данный момент является то, что валидационные датасеты не распределяются по workerам кластера, а после квантования хранятся целиком на мастере.\n",
    "\n",
    "Это надо учитывать и поэтому не стоит использовать большие валидационные датасеты, а также надо увеличивать `spark.driver.memoryOverhead` примерно на `1 * число_признаков * число_объектов_валидационного_датасета`. 1 тут - это потому что одно значение признака в квантованном датасете типично занимает 1 байт.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaab8fb8",
   "metadata": {},
   "source": [
    "## Обучение с категориальными признаками\n",
    "\n",
    "В Spark MLLib значения категориальных признаков стандартно так же лежат в DataFrame в том же массиве признаков типа `ml.linalg.Vector` как и численные и имеют целочисленные значения (хотя хранятся как double, так как `ml.linalg.Vector` может хранить только такие).\n",
    "\n",
    "Вышеописанная функция `createSchema` позволит нам правильно задать атрибуты в метаданных, так что нужные признаки будут распознаны как категориальные.\n",
    "\n",
    "Для разнообразия обучим на этот раз регрессионную, а не классификационную модель."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5d2db685",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee0ecea6ca0a43e38f387558feb29039",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-31:\n",
      "Traceback (most recent call last):\n",
      "  File \"/mnt/notebook-env/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/mnt/notebook-env/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 190, in cell_monitor\n",
      "    update_job_content(job_info_box, job_data)\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/updatewidgets.py\", line 38, in update_job_content\n",
      "    update_job_progress_bar(job_progress_bar_holder, job_data)\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/updatewidgets.py\", line 48, in update_job_progress_bar\n",
      "    progress_bar.max = new_max\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/traitlets/traitlets.py\", line 604, in __set__\n",
      "    self.set(obj, value)\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/traitlets/traitlets.py\", line 578, in set\n",
      "    new_value = self._validate(obj, value)\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/traitlets/traitlets.py\", line 612, in _validate\n",
      "    value = self._cross_validate(obj, value)\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/traitlets/traitlets.py\", line 618, in _cross_validate\n",
      "    value = obj._trait_validators[self.name](obj, proposal)\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/traitlets/traitlets.py\", line 975, in __call__\n",
      "    return self.func(*args, **kwargs)\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/ipywidgets/widgets/widget_int.py\", line 118, in _validate_max\n",
      "    raise TraitError('setting max < min')\n",
      "traitlets.traitlets.TraitError: setting max < min\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(features=DenseVector([0.0, 0.0, 0.0]), label=0.34, weight=1.0, prediction=0.2433638796338237)"
     ]
    }
   ],
   "source": [
    "srcDataSchema = createSchema(\n",
    "    [\n",
    "      (\"features\", VectorUDT()),\n",
    "      (\"label\", DoubleType()),\n",
    "      (\"weight\", FloatType())\n",
    "    ],\n",
    "    featureNames=[\"c1\", \"c2\", \"c3\"],\n",
    "    catFeaturesNumValues={\"c1\": 2, \"c2\": 4, \"c3\": 6},\n",
    "    addFeatureNamesMetadata=True\n",
    ")\n",
    "\n",
    "srcData = [\n",
    "  Row(Vectors.dense(0, 0, 0), 0.34, 1.0),\n",
    "  Row(Vectors.dense(1, 1, 0), 0.12, 0.12),\n",
    "  Row(Vectors.dense(0, 2, 1), 0.22, 0.18),\n",
    "  Row(Vectors.dense(1, 2, 2), 0.01, 1.0),\n",
    "  Row(Vectors.dense(0, 0, 3), 0.0, 2.0),\n",
    "  Row(Vectors.dense(0, 0, 4), 0.42, 0.45),\n",
    "  Row(Vectors.dense(1, 3, 5), 0.1, 1.0)\n",
    "]\n",
    "\n",
    "dfWithCatFeatures = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))\n",
    "\n",
    "regressor = catboost_spark.CatBoostRegressor(iterations=100)\n",
    "model_with_cat_features = regressor.fit(dfWithCatFeatures)\n",
    "prediction_with_cat_features = model_with_cat_features.transform(dfWithCatFeatures)\n",
    "prediction_with_cat_features.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43ebb5a7",
   "metadata": {},
   "source": [
    "Разумеется, возможны комбинации числовых и категориальных признаков в одном датасете."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "af0d21af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e61348b851f48f29687328bfdb77b54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(features=DenseVector([0.12, 0.0, 0.0]), label=0.34, weight=1.0, prediction=0.3002270728946602)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-32:\n",
      "Traceback (most recent call last):\n",
      "  File \"/mnt/notebook-env/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/mnt/notebook-env/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/mnt/notebook-env/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 178, in cell_monitor\n",
      "    job_binned_stages[job_id][stage_id] = all_stages[stage_id]\n",
      "KeyError: 1406\n",
      "\n"
     ]
    }
   ],
   "source": [
    "srcDataSchema = createSchema(\n",
    "    [\n",
    "      (\"features\", VectorUDT()),\n",
    "      (\"label\", DoubleType()),\n",
    "      (\"weight\", FloatType())\n",
    "    ],\n",
    "    featureNames=[\"f1\", \"c2\", \"c3\"],\n",
    "    catFeaturesNumValues={\"c2\": 4, \"c3\": 6},\n",
    "    addFeatureNamesMetadata=True\n",
    ")\n",
    "\n",
    "srcData = [\n",
    "  Row(Vectors.dense(0.12, 0, 0), 0.34, 1.0),\n",
    "  Row(Vectors.dense(0.1, 1, 0), 0.12, 0.12),\n",
    "  Row(Vectors.dense(0.0, 2, 1), 0.22, 0.18),\n",
    "  Row(Vectors.dense(0.33, 2, 2), 0.01, 1.0),\n",
    "  Row(Vectors.dense(0.8, 0, 3), 0.0, 2.0),\n",
    "  Row(Vectors.dense(0.0, 0, 4), 0.42, 0.45),\n",
    "  Row(Vectors.dense(0.01, 3, 5), 0.1, 1.0)\n",
    "]\n",
    "\n",
    "dfWithNumAndCatFeatures = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))\n",
    "\n",
    "regressor = catboost_spark.CatBoostRegressor(iterations=100)\n",
    "model_with_num_and_cat_features = regressor.fit(dfWithNumAndCatFeatures)\n",
    "prediction_with_num_and_cat_features = model_with_num_and_cat_features.transform(dfWithNumAndCatFeatures)\n",
    "prediction_with_num_and_cat_features.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b495cdf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PySpark",
   "language": "",
   "name": "pysparkkernel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "python",
    "version": 3
   },
   "mimetype": "text/x-python",
   "name": "pyspark",
   "pygments_lexer": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
